Add a run journal, progress modes and metrics for long runs - #79

Closes #46.

A publish run takes days, and when one finished short the only record of what happened was whatever scrolled past on stderr. Now every long command writes one JSON object per event to a journal file, so the two questions worth asking afterwards, which shards failed and when the rate dropped, are a jq filter away.

What is in it

One RunEvent schema covers the shard pipelines and the streaming commands, so a single query works across all of them. RunReporter fans each event out to three sinks: the journal on disk, stderr in text or JSON, and a Prometheus registry. A nil reporter is a working no-op, so library callers that never set one keep the stderr narration they have always had.

Three new global flags:

The --progress default is the useful one: run it in a terminal and you get human lines, redirect it into a log file or run it under a supervisor and you get JSON, without anybody having to remember a flag. The metrics address is bound at startup, so a busy port fails the command there rather than an hour in.

markdown export and markdown refetch emit start, per-shard, tick and end events with an inflight gauge, and default their journal to run.jsonl beside the ledger so a resumed run's events land next to the record of what it resumed from. download, export, host enrich and crawl seed report through StreamProgress, which counts items and phases on the same schedule and in the same shape. Those four reported nothing at all before.

Decisions worth flagging

The ticker reads the run stats through an atomic.Pointer snapshot, so it never takes the committer's lock. It recomputes rate and ETA from its own clock rather than from the stats, which only advance on a commit and so say nothing at exactly the moment you need them, when a run has stalled.

fetch_failed is a separate field from failed. A refetch shard that hit a thousand dead hosts still succeeded, and folding the two together would make a healthy run look broken.

The Prometheus registry is hand rolled. client_golang would do more of it, but it is a large dependency for twelve metrics in a binary whose selling point is that it has none, and the text exposition format is a dozen lines of printing.

RSS moves to build-tagged files: /proc/self/status on Linux, ps on macOS, 0 elsewhere. It was Linux only, which meant it always read 0 on the machine most of this gets developed on.

Also fixed

Smoke testing turned up an unrelated export bug. CDXStream wraps the callback's error with the page it came from, so the stop sentinel was compared against a wrapped error and never matched, and a perfectly successful ccrawl export 'example.com/*' -n 3 exited 1. It uses errors.Is now.

Done when

• one jq query over a finished run's run.jsonl answers which shards failed and when the rate dropped
• RSS is non zero on macOS

Verification

Real runs against Common Crawl on macOS, not just unit tests.

ccrawl export 'example.com/*' -n 3 --journal run.jsonl --progress json:

{"ts":"2026-08-07T06:16:41.304Z","event":"start","run":"20260807T061641Z-87559","pipeline":"export"}
{"ts":"2026-08-07T06:16:45.646Z","event":"end","run":"20260807T061641Z-87559","pipeline":"export","bytes":2861,"done":3,"rate_per_hour":2489.6,"elapsed_s":4.34,"rss_bytes":47267840}

Exit 0, and rss_bytes is 45 MB rather than the 0 it read before.

ccrawl download wet -n 4 -j 1 --metrics-addr 127.0.0.1:19091, scraped mid-run:

ccrawl_items_total{pipeline="download",status="ok"} 2
ccrawl_bytes_total{kind="fetched",pipeline="download"} 1.34689962e+08
ccrawl_done{pipeline="download"} 1
ccrawl_total{pipeline="download"} 4
ccrawl_rate_per_hour{pipeline="download"} 119.99505037216521
ccrawl_run_elapsed_seconds{pipeline="download"} 30.001237458
ccrawl_rss_bytes 2.8033024e+07

The matching tick carried an ETA:

{"event":"tick","pipeline":"download","bytes":65515836,"done":1,"total":4,"rate_per_hour":119.99,"eta_s":90.0,"elapsed_s":30.0,"rss_bytes":28033024}

go build ./..., go vet ./..., gofmt -l . and go test ./... are all clean, and scripts/docs-drift.sh passes with the three new flags documented.

Docs

New reference page at /reference/run-journal/ with the full event schema, the metric table, and the queries worth keeping. The three flags are in the global flags table.